{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dCpvgG0vwXAZ"
   },
   "source": [
    "# Predicting Movie Review Sentiment with BERT on TF Hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xiYrZKaHwV81"
   },
   "source": [
    "If you’ve been following Natural Language Processing over the past year, you’ve probably heard of BERT: Bidirectional Encoder Representations from Transformers. It’s a neural network architecture designed by Google researchers that’s totally transformed what’s state-of-the-art for NLP tasks, like text classification, translation, summarization, and question answering.\n",
    "\n",
    "Now that BERT's been added to [TF Hub](https://www.tensorflow.org/hub) as a loadable module, it's easy(ish) to add into existing Tensorflow text pipelines. In an existing pipeline, BERT can replace text embedding layers like ELMO and GloVE. Alternatively, [finetuning](http://wiki.fast.ai/index.php/Fine_tuning) BERT can provide both an accuracy boost and faster training time in many cases.\n",
    "\n",
    "Here, we'll train a model to predict whether a review is positive or negative using BERT in Tensorflow with tf hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Package                            Version   \r\n",
      "---------------------------------- ----------\r\n",
      "absl-py                            0.9.0     \r\n",
      "alabaster                          0.7.10    \r\n",
      "anaconda-client                    1.6.14    \r\n",
      "anaconda-project                   0.8.2     \r\n",
      "asn1crypto                         0.24.0    \r\n",
      "astor                              0.8.1     \r\n",
      "astroid                            1.6.3     \r\n",
      "astropy                            3.0.2     \r\n",
      "attrs                              18.1.0    \r\n",
      "Automat                            0.3.0     \r\n",
      "autovizwidget                      0.15.0    \r\n",
      "awscli                             1.18.20   \r\n",
      "Babel                              2.5.3     \r\n",
      "backcall                           0.1.0     \r\n",
      "backports.shutil-get-terminal-size 1.0.0     \r\n",
      "bcrypt                             3.1.7     \r\n",
      "beautifulsoup4                     4.6.0     \r\n",
      "bert-tensorflow                    1.0.1     \r\n",
      "bitarray                           0.8.1     \r\n",
      "bkcharts                           0.2       \r\n",
      "blaze                              0.11.3    \r\n",
      "bleach                             2.1.3     \r\n",
      "bokeh                              1.4.0     \r\n",
      "boto                               2.48.0    \r\n",
      "boto3                              1.12.20   \r\n",
      "botocore                           1.15.20   \r\n",
      "Bottleneck                         1.2.1     \r\n",
      "cached-property                    1.5.1     \r\n",
      "certifi                            2019.11.28\r\n",
      "cffi                               1.11.5    \r\n",
      "characteristic                     14.3.0    \r\n",
      "chardet                            3.0.4     \r\n",
      "click                              6.7       \r\n",
      "cloudpickle                        0.5.3     \r\n",
      "clyent                             1.2.2     \r\n",
      "colorama                           0.3.9     \r\n",
      "contextlib2                        0.5.5     \r\n",
      "cryptography                       2.8       \r\n",
      "cycler                             0.10.0    \r\n",
      "Cython                             0.28.4    \r\n",
      "cytoolz                            0.9.0.1   \r\n",
      "dask                               1.2.2     \r\n",
      "datashape                          0.5.4     \r\n",
      "decorator                          4.3.0     \r\n",
      "defusedxml                         0.6.0     \r\n",
      "distributed                        1.28.1    \r\n",
      "docker                             4.2.0     \r\n",
      "docker-compose                     1.25.4    \r\n",
      "dockerpty                          0.4.1     \r\n",
      "docopt                             0.6.2     \r\n",
      "docutils                           0.14      \r\n",
      "entrypoints                        0.2.3     \r\n",
      "enum34                             1.1.9     \r\n",
      "environment-kernels                1.1.1     \r\n",
      "et-xmlfile                         1.0.1     \r\n",
      "fastcache                          1.0.2     \r\n",
      "filelock                           3.0.4     \r\n",
      "Flask                              1.0.2     \r\n",
      "Flask-Cors                         3.0.4     \r\n",
      "gast                               0.2.2     \r\n",
      "gevent                             1.3.0     \r\n",
      "glob2                              0.6       \r\n",
      "gmpy2                              2.0.8     \r\n",
      "google-pasta                       0.1.8     \r\n",
      "greenlet                           0.4.13    \r\n",
      "grpcio                             1.10.1    \r\n",
      "h5py                               2.8.0     \r\n",
      "hdijupyterutils                    0.15.0    \r\n",
      "heapdict                           1.0.0     \r\n",
      "horovod                            0.19.0    \r\n",
      "html5lib                           1.0.1     \r\n",
      "idna                               2.6       \r\n",
      "imageio                            2.3.0     \r\n",
      "imagesize                          1.0.0     \r\n",
      "importlib-metadata                 1.5.0     \r\n",
      "ipykernel                          4.8.2     \r\n",
      "ipyparallel                        6.2.2     \r\n",
      "ipython                            6.4.0     \r\n",
      "ipython-genutils                   0.2.0     \r\n",
      "ipywidgets                         7.4.0     \r\n",
      "isort                              4.3.4     \r\n",
      "itsdangerous                       0.24      \r\n",
      "jdcal                              1.4       \r\n",
      "jedi                               0.12.0    \r\n",
      "Jinja2                             2.10      \r\n",
      "jmespath                           0.9.4     \r\n",
      "joblib                             0.14.1    \r\n",
      "jsonschema                         2.6.0     \r\n",
      "jupyter                            1.0.0     \r\n",
      "jupyter-client                     5.2.3     \r\n",
      "jupyter-console                    5.2.0     \r\n",
      "jupyter-core                       4.4.0     \r\n",
      "jupyterlab                         0.32.1    \r\n",
      "jupyterlab-launcher                0.10.5    \r\n",
      "Keras                              2.2.4     \r\n",
      "Keras-Applications                 1.0.8     \r\n",
      "Keras-Preprocessing                1.1.0     \r\n",
      "kiwisolver                         1.0.1     \r\n",
      "lazy-object-proxy                  1.3.1     \r\n",
      "llvmlite                           0.23.1    \r\n",
      "locket                             0.2.0     \r\n",
      "lxml                               4.2.1     \r\n",
      "Markdown                           3.2.1     \r\n",
      "MarkupSafe                         1.0       \r\n",
      "matplotlib                         3.0.3     \r\n",
      "mccabe                             0.6.1     \r\n",
      "mistune                            0.8.3     \r\n",
      "mkl-fft                            1.0.15    \r\n",
      "mkl-random                         1.1.0     \r\n",
      "mkl-service                        2.3.0     \r\n",
      "mock                               4.0.1     \r\n",
      "more-itertools                     4.1.0     \r\n",
      "mpmath                             1.0.0     \r\n",
      "msgpack                            0.6.0     \r\n",
      "msgpack-python                     0.5.6     \r\n",
      "multipledispatch                   0.5.0     \r\n",
      "nb-conda                           2.2.1     \r\n",
      "nb-conda-kernels                   2.2.2     \r\n",
      "nbconvert                          5.4.1     \r\n",
      "nbformat                           4.4.0     \r\n",
      "networkx                           2.1       \r\n",
      "nltk                               3.3       \r\n",
      "nose                               1.3.7     \r\n",
      "notebook                           5.5.0     \r\n",
      "numba                              0.38.0    \r\n",
      "numexpr                            2.7.1     \r\n",
      "numpy                              1.16.4    \r\n",
      "numpydoc                           0.8.0     \r\n",
      "odo                                0.5.1     \r\n",
      "olefile                            0.45.1    \r\n",
      "opencv-python                      3.4.2.17  \r\n",
      "openpyxl                           2.5.3     \r\n",
      "opt-einsum                         3.1.0     \r\n",
      "packaging                          20.1      \r\n",
      "pandas                             0.24.2    \r\n",
      "pandocfilters                      1.4.2     \r\n",
      "paramiko                           2.7.1     \r\n",
      "parso                              0.2.0     \r\n",
      "partd                              0.3.8     \r\n",
      "path.py                            11.0.1    \r\n",
      "pathlib2                           2.3.2     \r\n",
      "patsy                              0.5.0     \r\n",
      "pep8                               1.7.1     \r\n",
      "pexpect                            4.5.0     \r\n",
      "pickleshare                        0.7.4     \r\n",
      "Pillow                             5.4.1     \r\n",
      "pip                                19.3.1    \r\n",
      "pkginfo                            1.4.2     \r\n",
      "plotly                             4.5.2     \r\n",
      "pluggy                             0.6.0     \r\n",
      "ply                                3.11      \r\n",
      "prompt-toolkit                     1.0.15    \r\n",
      "protobuf                           3.8.0     \r\n",
      "protobuf3-to-dict                  0.1.5     \r\n",
      "psutil                             5.4.5     \r\n",
      "psycopg2                           2.7.5     \r\n",
      "ptyprocess                         0.5.2     \r\n",
      "py                                 1.5.3     \r\n",
      "py4j                               0.10.7    \r\n",
      "pyasn1                             0.4.8     \r\n",
      "pycodestyle                        2.4.0     \r\n",
      "pycosat                            0.6.3     \r\n",
      "pycparser                          2.18      \r\n",
      "pycrypto                           2.6.1     \r\n",
      "pycurl                             7.43.0.2  \r\n",
      "pyflakes                           1.6.0     \r\n",
      "pygal                              2.4.0     \r\n",
      "Pygments                           2.2.0     \r\n",
      "pykerberos                         1.2.1     \r\n",
      "pylint                             1.8.4     \r\n",
      "PyNaCl                             1.3.0     \r\n",
      "pyodbc                             4.0.23    \r\n",
      "pyOpenSSL                          18.0.0    \r\n",
      "pyparsing                          2.2.0     \r\n",
      "PySocks                            1.6.8     \r\n",
      "pyspark                            2.3.2     \r\n",
      "pytest                             3.5.1     \r\n",
      "pytest-arraydiff                   0.2       \r\n",
      "pytest-astropy                     0.3.0     \r\n",
      "pytest-doctestplus                 0.1.3     \r\n",
      "pytest-openfiles                   0.3.0     \r\n",
      "pytest-remotedata                  0.2.1     \r\n",
      "python-dateutil                    2.7.3     \r\n",
      "pytz                               2018.4    \r\n",
      "PyWavelets                         0.5.2     \r\n",
      "PyYAML                             5.3       \r\n",
      "pyzmq                              17.0.0    \r\n",
      "QtAwesome                          0.4.4     \r\n",
      "qtconsole                          4.3.1     \r\n",
      "QtPy                               1.4.1     \r\n",
      "requests                           2.20.0    \r\n",
      "requests-kerberos                  0.12.0    \r\n",
      "retrying                           1.3.3     \r\n",
      "rope                               0.10.7    \r\n",
      "rsa                                3.4.2     \r\n",
      "ruamel-yaml                        0.15.35   \r\n",
      "s3fs                               0.1.5     \r\n",
      "s3transfer                         0.3.3     \r\n",
      "sagemaker                          1.51.3    \r\n",
      "sagemaker-pyspark                  1.2.8     \r\n",
      "scikit-image                       0.13.1    \r\n",
      "scikit-learn                       0.22.1    \r\n",
      "scipy                              1.4.1     \r\n",
      "seaborn                            0.8.1     \r\n",
      "Send2Trash                         1.5.0     \r\n",
      "setuptools                         45.2.0    \r\n",
      "simplegeneric                      0.8.1     \r\n",
      "singledispatch                     3.4.0.3   \r\n",
      "six                                1.11.0    \r\n",
      "smdebug-rulesconfig                0.1.2     \r\n",
      "snowballstemmer                    1.2.1     \r\n",
      "sortedcollections                  0.6.1     \r\n",
      "sortedcontainers                   1.5.10    \r\n",
      "sparkmagic                         0.12.5    \r\n",
      "Sphinx                             1.7.4     \r\n",
      "sphinxcontrib-websupport           1.0.1     \r\n",
      "spyder                             3.2.8     \r\n",
      "SQLAlchemy                         1.2.11    \r\n",
      "statsmodels                        0.9.0     \r\n",
      "sympy                              1.1.1     \r\n",
      "tables                             3.4.3     \r\n",
      "TBB                                0.1       \r\n",
      "tblib                              1.3.2     \r\n",
      "tensorboard                        1.15.0    \r\n",
      "tensorflow                         1.15.2    \r\n",
      "tensorflow-estimator               1.15.1    \r\n",
      "tensorflow-hub                     0.7.0     \r\n",
      "tensorflow-serving-api             1.15.0    \r\n",
      "termcolor                          1.1.0     \r\n",
      "terminado                          0.8.1     \r\n",
      "testpath                           0.3.1     \r\n",
      "texttable                          1.6.2     \r\n",
      "toolz                              0.9.0     \r\n",
      "tornado                            5.0.2     \r\n",
      "traitlets                          4.3.2     \r\n",
      "typing                             3.6.4     \r\n",
      "unicodecsv                         0.14.1    \r\n",
      "urllib3                            1.23      \r\n",
      "wcwidth                            0.1.7     \r\n",
      "webencodings                       0.5.1     \r\n",
      "websocket-client                   0.57.0    \r\n",
      "Werkzeug                           0.14.1    \r\n",
      "wheel                              0.31.1    \r\n",
      "widgetsnbextension                 3.4.2     \r\n",
      "wrapt                              1.11.2    \r\n",
      "xlrd                               1.1.0     \r\n",
      "XlsxWriter                         1.0.4     \r\n",
      "xlwt                               1.3.0     \r\n",
      "zict                               0.1.3     \r\n",
      "zipp                               3.0.0     \r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorflow-hub in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (0.7.0)\n",
      "Requirement already satisfied: protobuf>=3.4.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-hub) (3.8.0)\n",
      "Requirement already satisfied: numpy>=1.12.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-hub) (1.16.4)\n",
      "Requirement already satisfied: six>=1.10.0 in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from tensorflow-hub) (1.11.0)\n",
      "Requirement already satisfied: setuptools in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from protobuf>=3.4.0->tensorflow-hub) (45.2.0)\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow-hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hsZvic2YxnTz",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/__init__.py:1467: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.15.2\n"
     ]
    }
   ],
   "source": [
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cp5wfXDx5SPH"
   },
   "source": [
    "In addition to the standard libraries we imported above, we'll need to install BERT's python package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "jviywGyWyKsA",
    "outputId": "166f3005-d219-404f-b201-2a0b75480360"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: bert-tensorflow in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (1.0.1)\n",
      "Requirement already satisfied: six in /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages (from bert-tensorflow) (1.11.0)\n",
      "\u001b[33mWARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install bert-tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hhbGEfwgdEtw"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/bert/optimization.py:87: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import bert\n",
    "from bert import run_classifier\n",
    "from bert import optimization\n",
    "from bert import tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KVB3eOcjxxm1"
   },
   "source": [
    "Below, we'll set an output directory location to store our model output and checkpoints. This can be a local directory, in which case you'd set OUTPUT_DIR to the name of the directory you'd like to create. If you're running this code in Google's hosted Colab, the directory won't persist after the Colab session ends.\n",
    "\n",
    "Alternatively, if you're a GCP user, you can store output in a GCP bucket. To do that, set a directory name in OUTPUT_DIR and the name of the GCP bucket in the BUCKET field.\n",
    "\n",
    "Set DO_DELETE to rewrite the OUTPUT_DIR if it exists. Otherwise, Tensorflow will load existing model checkpoints from that directory (if they exist)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "cellView": "form",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "US_EAnICvP7f",
    "outputId": "7780a032-31d4-4794-e6aa-664a5d2ae7dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** Model output directory: ./output *****\n"
     ]
    }
   ],
   "source": [
    "# Set the output directory for saving model file\n",
    "# Optionally, set a GCP bucket location\n",
    "\n",
    "OUTPUT_DIR = './output'#@param {type:\"string\"}\n",
    "#@markdown Whether or not to clear/delete the directory and create a new one\n",
    "DO_DELETE = True #@param {type:\"boolean\"}\n",
    "#@markdown Set USE_BUCKET and BUCKET if you want to (optionally) store model output on GCP bucket.\n",
    "USE_BUCKET = False #@param {type:\"boolean\"}\n",
    "BUCKET = 'BUCKET_NAME' #@param {type:\"string\"}\n",
    "\n",
    "if USE_BUCKET:\n",
    "  OUTPUT_DIR = 'gs://{}/{}'.format(BUCKET, OUTPUT_DIR)\n",
    "  from google.colab import auth\n",
    "  auth.authenticate_user()\n",
    "\n",
    "if DO_DELETE:\n",
    "  try:\n",
    "    tf.gfile.DeleteRecursively(OUTPUT_DIR)\n",
    "  except:\n",
    "    # Doesn't matter if the directory didn't exist\n",
    "    pass\n",
    "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
    "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pmFYvkylMwXn"
   },
   "source": [
    "#Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MC_w8SRqN0fr"
   },
   "source": [
    "First, let's download the dataset, hosted by Stanford. The code below, which downloads, extracts, and imports the IMDB Large Movie Review Dataset, is borrowed from [this Tensorflow tutorial](https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fom_ff20gyy6"
   },
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "import os\n",
    "import re\n",
    "\n",
    "# Load all files from a directory in a DataFrame.\n",
    "def load_directory_data(directory):\n",
    "  data = {}\n",
    "  data[\"sentence\"] = []\n",
    "  data[\"sentiment\"] = []\n",
    "  for file_path in os.listdir(directory):\n",
    "    with tf.gfile.GFile(os.path.join(directory, file_path), \"r\") as f:\n",
    "      data[\"sentence\"].append(f.read())\n",
    "      data[\"sentiment\"].append(re.match(\"\\d+_(\\d+)\\.txt\", file_path).group(1))\n",
    "  return pd.DataFrame.from_dict(data)\n",
    "\n",
    "# Merge positive and negative examples, add a polarity column and shuffle.\n",
    "def load_dataset(directory):\n",
    "  pos_df = load_directory_data(os.path.join(directory, \"pos\"))\n",
    "  neg_df = load_directory_data(os.path.join(directory, \"neg\"))\n",
    "  pos_df[\"polarity\"] = 1\n",
    "  neg_df[\"polarity\"] = 0\n",
    "  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "# Download and process the dataset files.\n",
    "def download_and_load_datasets(force_download=False):\n",
    "  dataset = tf.keras.utils.get_file(\n",
    "      fname=\"aclImdb.tar.gz\", \n",
    "      origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n",
    "      extract=True)\n",
    "  \n",
    "  train_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
    "                                       \"aclImdb\", \"train\"))\n",
    "  test_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
    "                                      \"aclImdb\", \"test\"))\n",
    "  \n",
    "  return train_df, test_df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2abfwdn-g135"
   },
   "outputs": [],
   "source": [
    "train, test = download_and_load_datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XA8WHJgzhIZf"
   },
   "source": [
    "To keep training fast, we'll take a sample of 5000 train and test examples, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lw_F488eixTV"
   },
   "outputs": [],
   "source": [
    "train = train.sample(5000)\n",
    "test = test.sample(5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>polarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8879</th>\n",
       "      <td>Well where do I begin my story?? I went to thi...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4386</th>\n",
       "      <td>Jake Speed is a film that lacks one thing  a ...</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6071</th>\n",
       "      <td>This early film has its flaws-- a predictable ...</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12488</th>\n",
       "      <td>Enchanting, romantic, innovative, and funny. T...</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21011</th>\n",
       "      <td>My interest in Dorothy Stratten caused me to p...</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                sentence sentiment  polarity\n",
       "8879   Well where do I begin my story?? I went to thi...         1         0\n",
       "4386   Jake Speed is a film that lacks one thing  a ...         7         1\n",
       "6071   This early film has its flaws-- a predictable ...         8         1\n",
       "12488  Enchanting, romantic, innovative, and funny. T...         9         1\n",
       "21011  My interest in Dorothy Stratten caused me to p...         4         0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>polarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20800</th>\n",
       "      <td>This almost perfect cinematic rendition of Edi...</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20320</th>\n",
       "      <td>I really don't understand who this movie is ai...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10500</th>\n",
       "      <td>I was greatly disappointed by the quality of t...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9957</th>\n",
       "      <td>I must admit I wasn't expecting much on this m...</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24085</th>\n",
       "      <td>So then... this is what passes as high art for...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                sentence sentiment  polarity\n",
       "20800  This almost perfect cinematic rendition of Edi...         8         1\n",
       "20320  I really don't understand who this movie is ai...         1         0\n",
       "10500  I was greatly disappointed by the quality of t...         2         0\n",
       "9957   I must admit I wasn't expecting much on this m...         8         1\n",
       "24085  So then... this is what passes as high art for...         1         0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "prRQM8pDi8xI",
    "outputId": "34445cb8-2be0-4379-fdbc-7794091f6049"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['sentence', 'sentiment', 'polarity'], dtype='object')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sfRnHSz3iSXz"
   },
   "source": [
    "For us, our input data is the 'sentence' column and our label is the 'polarity' column (0, 1 for negative and positive, respecitvely)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IuMOGwFui4it"
   },
   "outputs": [],
   "source": [
    "DATA_COLUMN = 'sentence'\n",
    "LABEL_COLUMN = 'polarity'\n",
    "# label_list is the list of labels, i.e. True, False or 0, 1 or 'dog', 'cat'\n",
    "label_list = [0, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V399W0rqNJ-Z"
   },
   "source": [
    "#Data Preprocessing\n",
    "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create  `InputExample`'s using the constructor provided in the BERT library.\n",
    "\n",
    "- `text_a` is the text we want to classify, which in this case, is the `Request` field in our Dataframe. \n",
    "- `text_b` is used if we're training a model to understand the relationship between sentences (i.e. is `text_b` a translation of `text_a`? Is `text_b` an answer to the question asked by `text_a`?). This doesn't apply to our task, so we can leave `text_b` blank.\n",
    "- `label` is the label for our example, i.e. True, False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p9gEt5SmM6i6"
   },
   "outputs": [],
   "source": [
    "# Use the InputExample class from BERT's run_classifier code to create examples from the data\n",
    "train_InputExamples = train.apply(lambda x: bert.run_classifier.InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
    "                                                                   text_a = x[DATA_COLUMN], \n",
    "                                                                   text_b = None, \n",
    "                                                                   label = x[LABEL_COLUMN]), axis = 1)\n",
    "\n",
    "test_InputExamples = test.apply(lambda x: bert.run_classifier.InputExample(guid=None, \n",
    "                                                                   text_a = x[DATA_COLUMN], \n",
    "                                                                   text_b = None, \n",
    "                                                                   label = x[LABEL_COLUMN]), axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SCZWZtKxObjh"
   },
   "source": [
    "Next, we need to preprocess our data so that it matches the data BERT was trained on. For this, we'll need to do a couple of things (but don't worry--this is also included in the Python library):\n",
    "\n",
    "\n",
    "1. Lowercase our text (if we're using a BERT lowercase model)\n",
    "2. Tokenize it (i.e. \"sally says hi\" -> [\"sally\", \"says\", \"hi\"])\n",
    "3. Break words into WordPieces (i.e. \"calling\" -> [\"call\", \"##ing\"])\n",
    "4. Map our words to indexes using a vocab file that BERT provides\n",
    "5. Add special \"CLS\" and \"SEP\" tokens (see the [readme](https://github.com/google-research/bert))\n",
    "6. Append \"index\" and \"segment\" tokens to each input (see the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))\n",
    "\n",
    "Happily, we don't have to worry about most of these details.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qMWiDtpyQSoU"
   },
   "source": [
    "To start, we'll need to load a vocabulary file and lowercasing information directly from the BERT tf hub module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "IhJSe0QHNG7U",
    "outputId": "20b28cc7-3cb3-4ce6-bfff-a7847ce3bbaa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    }
   ],
   "source": [
    "# This is a path to an uncased (all lowercase) version of BERT\n",
    "BERT_MODEL_HUB = \"https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1\"\n",
    "\n",
    "def create_tokenizer_from_hub_module():\n",
    "  \"\"\"Get the vocab file and casing info from the Hub module.\"\"\"\n",
    "  with tf.Graph().as_default():\n",
    "    bert_module = hub.Module(BERT_MODEL_HUB)\n",
    "    tokenization_info = bert_module(signature=\"tokenization_info\", as_dict=True)\n",
    "    with tf.Session() as sess:\n",
    "      vocab_file, do_lower_case = sess.run([tokenization_info[\"vocab_file\"],\n",
    "                                            tokenization_info[\"do_lower_case\"]])\n",
    "      \n",
    "  return bert.tokenization.FullTokenizer(\n",
    "      vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
    "\n",
    "tokenizer = create_tokenizer_from_hub_module()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z4oFkhpZBDKm"
   },
   "source": [
    "Great--we just learned that the BERT model we're using expects lowercase data (that's what stored in tokenization_info[\"do_lower_case\"]) and we also loaded BERT's vocab file. We also created a tokenizer, which breaks words into word pieces:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "dsBo6RCtQmwx",
    "outputId": "9af8c917-90ec-4fe9-897b-79dc89ca88e1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['this',\n",
       " 'here',\n",
       " \"'\",\n",
       " 's',\n",
       " 'an',\n",
       " 'example',\n",
       " 'of',\n",
       " 'using',\n",
       " 'the',\n",
       " 'bert',\n",
       " 'token',\n",
       " '##izer']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.tokenize(\"This here's an example of using the BERT tokenizer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0OEzfFIt6GIc"
   },
   "source": [
    "Using our tokenizer, we'll call `run_classifier.convert_examples_to_features` on our InputExamples to convert them into features BERT understands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll set sequences to be at most 128 tokens long.\n",
    "MAX_SEQ_LENGTH = 128\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1261
    },
    "colab_type": "code",
    "id": "LL5W8gEGRTAf",
    "outputId": "65001dda-155b-48fc-b5fc-1e4cabc8dfbf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/bert/run_classifier.py:774: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/bert/run_classifier.py:774: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 5000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 5000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] well where do i begin my story ? ? i went to this movie tonight with a few friends not knowing more than the actors that were in it , and that it was supposed to be a horror movie . < br / > < br / > well i figured out within the first 20 minutes , what a poor decision i had made going out seeing this movie . the plot was crap , and so was the script . the lines were horrible to the point that people in the audience were laughing hysterical ##ly . < br / > < br / > the cast couldn ' t have been more plastic looking . even some of the scenes seemed like [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] well where do i begin my story ? ? i went to this movie tonight with a few friends not knowing more than the actors that were in it , and that it was supposed to be a horror movie . < br / > < br / > well i figured out within the first 20 minutes , what a poor decision i had made going out seeing this movie . the plot was crap , and so was the script . the lines were horrible to the point that people in the audience were laughing hysterical ##ly . < br / > < br / > the cast couldn ' t have been more plastic looking . even some of the scenes seemed like [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2092 2073 2079 1045 4088 2026 2466 1029 1029 1045 2253 2000 2023 3185 3892 2007 1037 2261 2814 2025 4209 2062 2084 1996 5889 2008 2020 1999 2009 1010 1998 2008 2009 2001 4011 2000 2022 1037 5469 3185 1012 1026 7987 1013 1028 1026 7987 1013 1028 2092 1045 6618 2041 2306 1996 2034 2322 2781 1010 2054 1037 3532 3247 1045 2018 2081 2183 2041 3773 2023 3185 1012 1996 5436 2001 10231 1010 1998 2061 2001 1996 5896 1012 1996 3210 2020 9202 2000 1996 2391 2008 2111 1999 1996 4378 2020 5870 25614 2135 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 3459 2481 1005 1056 2031 2042 2062 6081 2559 1012 2130 2070 1997 1996 5019 2790 2066 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2092 2073 2079 1045 4088 2026 2466 1029 1029 1045 2253 2000 2023 3185 3892 2007 1037 2261 2814 2025 4209 2062 2084 1996 5889 2008 2020 1999 2009 1010 1998 2008 2009 2001 4011 2000 2022 1037 5469 3185 1012 1026 7987 1013 1028 1026 7987 1013 1028 2092 1045 6618 2041 2306 1996 2034 2322 2781 1010 2054 1037 3532 3247 1045 2018 2081 2183 2041 3773 2023 3185 1012 1996 5436 2001 10231 1010 1998 2061 2001 1996 5896 1012 1996 3210 2020 9202 2000 1996 2391 2008 2111 1999 1996 4378 2020 5870 25614 2135 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 3459 2481 1005 1056 2031 2042 2062 6081 2559 1012 2130 2070 1997 1996 5019 2790 2066 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] jake speed is a film that lacks one thing a charismatic lead . unfortunately that ' s something that really tai ##nts the entire movie and it ' s a shame because at heart it is an enjoyable action movie with a witty enough script and an interesting , if derivative , premise . although it ' s genesis probably can be traced back to the success of the indiana jones trilogy the film actually plays a little more like ' roman ##cing the stone ' albeit in reverse . it ' s not an author of romantic adventure fiction being led on an adventure by a character very much like one of her creations it is an adventure fiction character ( who happens to chronicle [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] jake speed is a film that lacks one thing a charismatic lead . unfortunately that ' s something that really tai ##nts the entire movie and it ' s a shame because at heart it is an enjoyable action movie with a witty enough script and an interesting , if derivative , premise . although it ' s genesis probably can be traced back to the success of the indiana jones trilogy the film actually plays a little more like ' roman ##cing the stone ' albeit in reverse . it ' s not an author of romantic adventure fiction being led on an adventure by a character very much like one of her creations it is an adventure fiction character ( who happens to chronicle [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 5180 3177 2003 1037 2143 2008 14087 2028 2518 1037 23916 2599 1012 6854 2008 1005 1055 2242 2008 2428 13843 7666 1996 2972 3185 1998 2009 1005 1055 1037 9467 2138 2012 2540 2009 2003 2019 22249 2895 3185 2007 1037 25591 2438 5896 1998 2019 5875 1010 2065 13819 1010 18458 1012 2348 2009 1005 1055 11046 2763 2064 2022 9551 2067 2000 1996 3112 1997 1996 5242 3557 11544 1996 2143 2941 3248 1037 2210 2062 2066 1005 3142 6129 1996 2962 1005 12167 1999 7901 1012 2009 1005 1055 2025 2019 3166 1997 6298 6172 4349 2108 2419 2006 2019 6172 2011 1037 2839 2200 2172 2066 2028 1997 2014 20677 2009 2003 2019 6172 4349 2839 1006 2040 6433 2000 9519 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 5180 3177 2003 1037 2143 2008 14087 2028 2518 1037 23916 2599 1012 6854 2008 1005 1055 2242 2008 2428 13843 7666 1996 2972 3185 1998 2009 1005 1055 1037 9467 2138 2012 2540 2009 2003 2019 22249 2895 3185 2007 1037 25591 2438 5896 1998 2019 5875 1010 2065 13819 1010 18458 1012 2348 2009 1005 1055 11046 2763 2064 2022 9551 2067 2000 1996 3112 1997 1996 5242 3557 11544 1996 2143 2941 3248 1037 2210 2062 2066 1005 3142 6129 1996 2962 1005 12167 1999 7901 1012 2009 1005 1055 2025 2019 3166 1997 6298 6172 4349 2108 2419 2006 2019 6172 2011 1037 2839 2200 2172 2066 2028 1997 2014 20677 2009 2003 2019 6172 4349 2839 1006 2040 6433 2000 9519 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] this early film has its flaws - - a predictable plot and some over ##long scenes of dubious relevance - - but it already clearly demonstrates hitchcock ' s mastery of editing and the use of powerful images . it ' s also among the most expression ##ist of his films stylistic ##ally ; note , for examples , the weird distortion ##s he uses during the party sequence and the frequent echoes of both title and plot in the imagery . < br / > < br / > its core , though , remains the final match , which is still among the more exciting examples of cinematic boxing . even though you know that the hero has to win , it becomes quite [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] this early film has its flaws - - a predictable plot and some over ##long scenes of dubious relevance - - but it already clearly demonstrates hitchcock ' s mastery of editing and the use of powerful images . it ' s also among the most expression ##ist of his films stylistic ##ally ; note , for examples , the weird distortion ##s he uses during the party sequence and the frequent echoes of both title and plot in the imagery . < br / > < br / > its core , though , remains the final match , which is still among the more exciting examples of cinematic boxing . even though you know that the hero has to win , it becomes quite [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2023 2220 2143 2038 2049 21407 1011 1011 1037 21425 5436 1998 2070 2058 10052 5019 1997 22917 21923 1011 1011 2021 2009 2525 4415 16691 19625 1005 1055 26364 1997 9260 1998 1996 2224 1997 3928 4871 1012 2009 1005 1055 2036 2426 1996 2087 3670 2923 1997 2010 3152 24828 3973 1025 3602 1010 2005 4973 1010 1996 6881 20870 2015 2002 3594 2076 1996 2283 5537 1998 1996 6976 17659 1997 2119 2516 1998 5436 1999 1996 13425 1012 1026 7987 1013 1028 1026 7987 1013 1028 2049 4563 1010 2295 1010 3464 1996 2345 2674 1010 2029 2003 2145 2426 1996 2062 10990 4973 1997 21014 8362 1012 2130 2295 2017 2113 2008 1996 5394 2038 2000 2663 1010 2009 4150 3243 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2023 2220 2143 2038 2049 21407 1011 1011 1037 21425 5436 1998 2070 2058 10052 5019 1997 22917 21923 1011 1011 2021 2009 2525 4415 16691 19625 1005 1055 26364 1997 9260 1998 1996 2224 1997 3928 4871 1012 2009 1005 1055 2036 2426 1996 2087 3670 2923 1997 2010 3152 24828 3973 1025 3602 1010 2005 4973 1010 1996 6881 20870 2015 2002 3594 2076 1996 2283 5537 1998 1996 6976 17659 1997 2119 2516 1998 5436 1999 1996 13425 1012 1026 7987 1013 1028 1026 7987 1013 1028 2049 4563 1010 2295 1010 3464 1996 2345 2674 1010 2029 2003 2145 2426 1996 2062 10990 4973 1997 21014 8362 1012 2130 2295 2017 2113 2008 1996 5394 2038 2000 2663 1010 2009 4150 3243 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] en ##chan ##ting , romantic , innovative , and funny . the vision of this extraordinary film is almost un ##para ##lle ##led , exceeding better known \" death romance ##s \" such as ghost . while we know intuitive ##ly that peter and june will find ultimate happiness at the end of that long - long stairway , the joy is in the journey . the moral of the tale , of course , is timeless : love conquer ##s all . but the struggle to achieve that victory is played in a celestial arena of sweeping vision and gripping grande ##ur . with more than 500 suit ##ably clad extras portraying various ages and cultures , the directors ' vision of heaven remains memorable [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] en ##chan ##ting , romantic , innovative , and funny . the vision of this extraordinary film is almost un ##para ##lle ##led , exceeding better known \" death romance ##s \" such as ghost . while we know intuitive ##ly that peter and june will find ultimate happiness at the end of that long - long stairway , the joy is in the journey . the moral of the tale , of course , is timeless : love conquer ##s all . but the struggle to achieve that victory is played in a celestial arena of sweeping vision and gripping grande ##ur . with more than 500 suit ##ably clad extras portraying various ages and cultures , the directors ' vision of heaven remains memorable [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 4372 14856 3436 1010 6298 1010 9525 1010 1998 6057 1012 1996 4432 1997 2023 9313 2143 2003 2471 4895 28689 6216 3709 1010 17003 2488 2124 1000 2331 7472 2015 1000 2107 2004 5745 1012 2096 2057 2113 29202 2135 2008 2848 1998 2238 2097 2424 7209 8404 2012 1996 2203 1997 2008 2146 1011 2146 21952 1010 1996 6569 2003 1999 1996 4990 1012 1996 7191 1997 1996 6925 1010 1997 2607 1010 2003 27768 1024 2293 16152 2015 2035 1012 2021 1996 5998 2000 6162 2008 3377 2003 2209 1999 1037 17617 5196 1997 12720 4432 1998 13940 9026 3126 1012 2007 2062 2084 3156 4848 8231 13681 26279 17274 2536 5535 1998 8578 1010 1996 5501 1005 4432 1997 6014 3464 13432 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 4372 14856 3436 1010 6298 1010 9525 1010 1998 6057 1012 1996 4432 1997 2023 9313 2143 2003 2471 4895 28689 6216 3709 1010 17003 2488 2124 1000 2331 7472 2015 1000 2107 2004 5745 1012 2096 2057 2113 29202 2135 2008 2848 1998 2238 2097 2424 7209 8404 2012 1996 2203 1997 2008 2146 1011 2146 21952 1010 1996 6569 2003 1999 1996 4990 1012 1996 7191 1997 1996 6925 1010 1997 2607 1010 2003 27768 1024 2293 16152 2015 2035 1012 2021 1996 5998 2000 6162 2008 3377 2003 2209 1999 1037 17617 5196 1997 12720 4432 1998 13940 9026 3126 1012 2007 2062 2084 3156 4848 8231 13681 26279 17274 2536 5535 1998 8578 1010 1996 5501 1005 4432 1997 6014 3464 13432 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] my interest in dorothy st ##rat ##ten caused me to purchase this video . although it had great actors / actresses , there were just too many sub ##pl ##ots going on to retain interest . plus it just wasn ' t that interesting . dialogue was stiff and confusing and the story just flipped around too much to be bel ##ie ##vable . i was pretty disappointed in what i believe was one of audrey hepburn ' s last movies . i ' ll always love john ritter best in slap ##stick . he was just too pathetic here . [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] my interest in dorothy st ##rat ##ten caused me to purchase this video . although it had great actors / actresses , there were just too many sub ##pl ##ots going on to retain interest . plus it just wasn ' t that interesting . dialogue was stiff and confusing and the story just flipped around too much to be bel ##ie ##vable . i was pretty disappointed in what i believe was one of audrey hepburn ' s last movies . i ' ll always love john ritter best in slap ##stick . he was just too pathetic here . [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2026 3037 1999 9984 2358 8609 6528 3303 2033 2000 5309 2023 2678 1012 2348 2009 2018 2307 5889 1013 19910 1010 2045 2020 2074 2205 2116 4942 24759 12868 2183 2006 2000 9279 3037 1012 4606 2009 2074 2347 1005 1056 2008 5875 1012 7982 2001 10551 1998 16801 1998 1996 2466 2074 9357 2105 2205 2172 2000 2022 19337 2666 12423 1012 1045 2001 3492 9364 1999 2054 1045 2903 2001 2028 1997 14166 22004 1005 1055 2197 5691 1012 1045 1005 2222 2467 2293 2198 23168 2190 1999 14308 21354 1012 2002 2001 2074 2205 17203 2182 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2026 3037 1999 9984 2358 8609 6528 3303 2033 2000 5309 2023 2678 1012 2348 2009 2018 2307 5889 1013 19910 1010 2045 2020 2074 2205 2116 4942 24759 12868 2183 2006 2000 9279 3037 1012 4606 2009 2074 2347 1005 1056 2008 5875 1012 7982 2001 10551 1998 16801 1998 1996 2466 2074 9357 2105 2205 2172 2000 2022 19337 2666 12423 1012 1045 2001 3492 9364 1999 2054 1045 2903 2001 2028 1997 14166 22004 1005 1055 2197 5691 1012 1045 1005 2222 2467 2293 2198 23168 2190 1999 14308 21354 1012 2002 2001 2074 2205 17203 2182 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 5000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Writing example 0 of 5000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] this almost perfect cinematic rendition of edith nes ##bit ' s popular children ' s novel follows the lives of roberta ( bobbie ) , phyllis , and peter , and their mother , after their father is unfair ##ly accused of treason and sent to prison . they go to live in an almost un ##in ##hab ##ita ##ble house in the country which stands near a railway line mum writes stories to make enough money for food and candles , while the children spend much of their time around the railway station and , specifically , waving to one particular train to ' send their love to father ' . < br / > < br / > always an involving and clever novel [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] this almost perfect cinematic rendition of edith nes ##bit ' s popular children ' s novel follows the lives of roberta ( bobbie ) , phyllis , and peter , and their mother , after their father is unfair ##ly accused of treason and sent to prison . they go to live in an almost un ##in ##hab ##ita ##ble house in the country which stands near a railway line mum writes stories to make enough money for food and candles , while the children spend much of their time around the railway station and , specifically , waving to one particular train to ' send their love to father ' . < br / > < br / > always an involving and clever novel [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2023 2471 3819 21014 19187 1997 13257 24524 16313 1005 1055 2759 2336 1005 1055 3117 4076 1996 3268 1997 23455 1006 27731 1007 1010 20328 1010 1998 2848 1010 1998 2037 2388 1010 2044 2037 2269 2003 15571 2135 5496 1997 14712 1998 2741 2000 3827 1012 2027 2175 2000 2444 1999 2019 2471 4895 2378 25459 6590 3468 2160 1999 1996 2406 2029 4832 2379 1037 2737 2240 12954 7009 3441 2000 2191 2438 2769 2005 2833 1998 14006 1010 2096 1996 2336 5247 2172 1997 2037 2051 2105 1996 2737 2276 1998 1010 4919 1010 12015 2000 2028 3327 3345 2000 1005 4604 2037 2293 2000 2269 1005 1012 1026 7987 1013 1028 1026 7987 1013 1028 2467 2019 5994 1998 12266 3117 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2023 2471 3819 21014 19187 1997 13257 24524 16313 1005 1055 2759 2336 1005 1055 3117 4076 1996 3268 1997 23455 1006 27731 1007 1010 20328 1010 1998 2848 1010 1998 2037 2388 1010 2044 2037 2269 2003 15571 2135 5496 1997 14712 1998 2741 2000 3827 1012 2027 2175 2000 2444 1999 2019 2471 4895 2378 25459 6590 3468 2160 1999 1996 2406 2029 4832 2379 1037 2737 2240 12954 7009 3441 2000 2191 2438 2769 2005 2833 1998 14006 1010 2096 1996 2336 5247 2172 1997 2037 2051 2105 1996 2737 2276 1998 1010 4919 1010 12015 2000 2028 3327 3345 2000 1005 4604 2037 2293 2000 2269 1005 1012 1026 7987 1013 1028 1026 7987 1013 1028 2467 2019 5994 1998 12266 3117 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] i really don ' t understand who this movie is aimed at . from just the absurd ##ity of it , not to mention the ridiculous ##ly bad acting , che ##es ##y dialogue , and the fact that the villain is a child , i ' d assume this was meant to be a children ' s movie . . . but i think there may be more swear words than pulp fiction , not to mention constant references to drugs and general mayhem and killing - so which demographic is it trying to please ? this movie is too sc ##hi ##zo ##ph ##ren ##ic , like trying to combine country music with heavy metal , in the end no one is going to [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] i really don ' t understand who this movie is aimed at . from just the absurd ##ity of it , not to mention the ridiculous ##ly bad acting , che ##es ##y dialogue , and the fact that the villain is a child , i ' d assume this was meant to be a children ' s movie . . . but i think there may be more swear words than pulp fiction , not to mention constant references to drugs and general mayhem and killing - so which demographic is it trying to please ? this movie is too sc ##hi ##zo ##ph ##ren ##ic , like trying to combine country music with heavy metal , in the end no one is going to [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1045 2428 2123 1005 1056 3305 2040 2023 3185 2003 6461 2012 1012 2013 2074 1996 18691 3012 1997 2009 1010 2025 2000 5254 1996 9951 2135 2919 3772 1010 18178 2229 2100 7982 1010 1998 1996 2755 2008 1996 12700 2003 1037 2775 1010 1045 1005 1040 7868 2023 2001 3214 2000 2022 1037 2336 1005 1055 3185 1012 1012 1012 2021 1045 2228 2045 2089 2022 2062 8415 2616 2084 16016 4349 1010 2025 2000 5254 5377 7604 2000 5850 1998 2236 26865 1998 4288 1011 2061 2029 15982 2003 2009 2667 2000 3531 1029 2023 3185 2003 2205 8040 4048 6844 8458 7389 2594 1010 2066 2667 2000 11506 2406 2189 2007 3082 3384 1010 1999 1996 2203 2053 2028 2003 2183 2000 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1045 2428 2123 1005 1056 3305 2040 2023 3185 2003 6461 2012 1012 2013 2074 1996 18691 3012 1997 2009 1010 2025 2000 5254 1996 9951 2135 2919 3772 1010 18178 2229 2100 7982 1010 1998 1996 2755 2008 1996 12700 2003 1037 2775 1010 1045 1005 1040 7868 2023 2001 3214 2000 2022 1037 2336 1005 1055 3185 1012 1012 1012 2021 1045 2228 2045 2089 2022 2062 8415 2616 2084 16016 4349 1010 2025 2000 5254 5377 7604 2000 5850 1998 2236 26865 1998 4288 1011 2061 2029 15982 2003 2009 2667 2000 3531 1029 2023 3185 2003 2205 8040 4048 6844 8458 7389 2594 1010 2066 2667 2000 11506 2406 2189 2007 3082 3384 1010 1999 1996 2203 2053 2028 2003 2183 2000 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] i was greatly disappointed by the quality of this documentary . the content is poorly produced , very poor quality video and , especially awful audio . there ' s extremely little about how bruce ha ##ack produced his music and virtually no examples of direct connection to later and contemporary electronic music . the interviews of people who knew bruce ha ##ack are ad - hoc mostly ina ##rti ##cula ##te mum ##bo - ju ##mbo . too much ya ##k and not enough ha ##ack . although i have a serious personal interest in electronic music and have a higher than average attention span , even for slow and / or difficult subject matter , i fell asleep while watching this documentary and had [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] i was greatly disappointed by the quality of this documentary . the content is poorly produced , very poor quality video and , especially awful audio . there ' s extremely little about how bruce ha ##ack produced his music and virtually no examples of direct connection to later and contemporary electronic music . the interviews of people who knew bruce ha ##ack are ad - hoc mostly ina ##rti ##cula ##te mum ##bo - ju ##mbo . too much ya ##k and not enough ha ##ack . although i have a serious personal interest in electronic music and have a higher than average attention span , even for slow and / or difficult subject matter , i fell asleep while watching this documentary and had [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1045 2001 6551 9364 2011 1996 3737 1997 2023 4516 1012 1996 4180 2003 9996 2550 1010 2200 3532 3737 2678 1998 1010 2926 9643 5746 1012 2045 1005 1055 5186 2210 2055 2129 5503 5292 8684 2550 2010 2189 1998 8990 2053 4973 1997 3622 4434 2000 2101 1998 3824 4816 2189 1012 1996 7636 1997 2111 2040 2354 5503 5292 8684 2024 4748 1011 21929 3262 27118 28228 19879 2618 12954 5092 1011 18414 13344 1012 2205 2172 8038 2243 1998 2025 2438 5292 8684 1012 2348 1045 2031 1037 3809 3167 3037 1999 4816 2189 1998 2031 1037 3020 2084 2779 3086 8487 1010 2130 2005 4030 1998 1013 2030 3697 3395 3043 1010 1045 3062 6680 2096 3666 2023 4516 1998 2018 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1045 2001 6551 9364 2011 1996 3737 1997 2023 4516 1012 1996 4180 2003 9996 2550 1010 2200 3532 3737 2678 1998 1010 2926 9643 5746 1012 2045 1005 1055 5186 2210 2055 2129 5503 5292 8684 2550 2010 2189 1998 8990 2053 4973 1997 3622 4434 2000 2101 1998 3824 4816 2189 1012 1996 7636 1997 2111 2040 2354 5503 5292 8684 2024 4748 1011 21929 3262 27118 28228 19879 2618 12954 5092 1011 18414 13344 1012 2205 2172 8038 2243 1998 2025 2438 5292 8684 1012 2348 1045 2031 1037 3809 3167 3037 1999 4816 2189 1998 2031 1037 3020 2084 2779 3086 8487 1010 2130 2005 4030 1998 1013 2030 3697 3395 3043 1010 1045 3062 6680 2096 3666 2023 4516 1998 2018 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] i must admit i wasn ' t expecting much on this movie . i was surprised i truly enjoyed it as much as i did . the script wasn ' t oscar material , but it wasn ' t horrible either . the acting was great by mark wah ##lberg . jennifer an ##isto ##n had a great supporting role , and looked lovely as ever . what made this movie for me was the music . if you do not like 80 ' s g ##lam metal or hair bands , then you probably won ##t like this movie . its all about being a rocks ##tar . some cl ##iche ' s were present , but didn ' t bring down the movie at [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] i must admit i wasn ' t expecting much on this movie . i was surprised i truly enjoyed it as much as i did . the script wasn ' t oscar material , but it wasn ' t horrible either . the acting was great by mark wah ##lberg . jennifer an ##isto ##n had a great supporting role , and looked lovely as ever . what made this movie for me was the music . if you do not like 80 ' s g ##lam metal or hair bands , then you probably won ##t like this movie . its all about being a rocks ##tar . some cl ##iche ' s were present , but didn ' t bring down the movie at [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1045 2442 6449 1045 2347 1005 1056 8074 2172 2006 2023 3185 1012 1045 2001 4527 1045 5621 5632 2009 2004 2172 2004 1045 2106 1012 1996 5896 2347 1005 1056 7436 3430 1010 2021 2009 2347 1005 1056 9202 2593 1012 1996 3772 2001 2307 2011 2928 22894 22927 1012 7673 2019 20483 2078 2018 1037 2307 4637 2535 1010 1998 2246 8403 2004 2412 1012 2054 2081 2023 3185 2005 2033 2001 1996 2189 1012 2065 2017 2079 2025 2066 3770 1005 1055 1043 10278 3384 2030 2606 4996 1010 2059 2017 2763 2180 2102 2066 2023 3185 1012 2049 2035 2055 2108 1037 5749 7559 1012 2070 18856 17322 1005 1055 2020 2556 1010 2021 2134 1005 1056 3288 2091 1996 3185 2012 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 1045 2442 6449 1045 2347 1005 1056 8074 2172 2006 2023 3185 1012 1045 2001 4527 1045 5621 5632 2009 2004 2172 2004 1045 2106 1012 1996 5896 2347 1005 1056 7436 3430 1010 2021 2009 2347 1005 1056 9202 2593 1012 1996 3772 2001 2307 2011 2928 22894 22927 1012 7673 2019 20483 2078 2018 1037 2307 4637 2535 1010 1998 2246 8403 2004 2412 1012 2054 2081 2023 3185 2005 2033 2001 1996 2189 1012 2065 2017 2079 2025 2066 3770 1005 1055 1043 10278 3384 2030 2606 4996 1010 2059 2017 2763 2180 2102 2066 2023 3185 1012 2049 2035 2055 2108 1037 5749 7559 1012 2070 18856 17322 1005 1055 2020 2556 1010 2021 2134 1005 1056 3288 2091 1996 3185 2012 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 1 (id = 1)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:*** Example ***\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:guid: None\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] so then . . . this is what passes as high art for the likes of s ##x ##sw film festival and sundance , eh ? well , i suppose i can relate as long as story , script , dial ##og , acting ( save for ms . as ##elt ##on ) , cinematography and editing are completely irrelevant . < br / > < br / > i remember telling other film - making friends some years ago that the biggest problem with digital video was that we were now going to have to wade through a future sea of crap to get to anything worth watching now that anyone and his brother ( or brothers in the case of the du ##pl ##ass [SEP]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:tokens: [CLS] so then . . . this is what passes as high art for the likes of s ##x ##sw film festival and sundance , eh ? well , i suppose i can relate as long as story , script , dial ##og , acting ( save for ms . as ##elt ##on ) , cinematography and editing are completely irrelevant . < br / > < br / > i remember telling other film - making friends some years ago that the biggest problem with digital video was that we were now going to have to wade through a future sea of crap to get to anything worth watching now that anyone and his brother ( or brothers in the case of the du ##pl ##ass [SEP]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2061 2059 1012 1012 1012 2023 2003 2054 5235 2004 2152 2396 2005 1996 7777 1997 1055 2595 26760 2143 2782 1998 20140 1010 15501 1029 2092 1010 1045 6814 1045 2064 14396 2004 2146 2004 2466 1010 5896 1010 13764 8649 1010 3772 1006 3828 2005 5796 1012 2004 20042 2239 1007 1010 16434 1998 9260 2024 3294 22537 1012 1026 7987 1013 1028 1026 7987 1013 1028 1045 3342 4129 2060 2143 1011 2437 2814 2070 2086 3283 2008 1996 5221 3291 2007 3617 2678 2001 2008 2057 2020 2085 2183 2000 2031 2000 10653 2083 1037 2925 2712 1997 10231 2000 2131 2000 2505 4276 3666 2085 2008 3087 1998 2010 2567 1006 2030 3428 1999 1996 2553 1997 1996 4241 24759 12054 102\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_ids: 101 2061 2059 1012 1012 1012 2023 2003 2054 5235 2004 2152 2396 2005 1996 7777 1997 1055 2595 26760 2143 2782 1998 20140 1010 15501 1029 2092 1010 1045 6814 1045 2064 14396 2004 2146 2004 2466 1010 5896 1010 13764 8649 1010 3772 1006 3828 2005 5796 1012 2004 20042 2239 1007 1010 16434 1998 9260 2024 3294 22537 1012 1026 7987 1013 1028 1026 7987 1013 1028 1045 3342 4129 2060 2143 1011 2437 2814 2070 2086 3283 2008 1996 5221 3291 2007 3617 2678 2001 2008 2057 2020 2085 2183 2000 2031 2000 10653 2083 1037 2925 2712 1997 10231 2000 2131 2000 2505 4276 3666 2085 2008 3087 1998 2010 2567 1006 2030 3428 1999 1996 2553 1997 1996 4241 24759 12054 102\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:label: 0 (id = 0)\n"
     ]
    }
   ],
   "source": [
    "# Convert our train and test features to InputFeatures that BERT understands.\n",
    "train_features = bert.run_classifier.convert_examples_to_features(train_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
    "test_features = bert.run_classifier.convert_examples_to_features(test_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ccp5trMwRtmr"
   },
   "source": [
    "#Creating a model\n",
    "\n",
    "Now that we've prepared our data, let's focus on building a model. `create_model` does just this below. First, it loads the BERT tf hub module again (this time to extract the computation graph). Next, it creates a single new layer that will be trained to adapt BERT to our sentiment task (i.e. classifying whether a movie review is positive or negative). This strategy of using a mostly trained model is called [fine-tuning](http://wiki.fast.ai/index.php/Fine_tuning)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6o2a5ZIvRcJq"
   },
   "outputs": [],
   "source": [
    "def create_model(is_predicting, input_ids, input_mask, segment_ids, labels,\n",
    "                 num_labels):\n",
    "  \"\"\"Creates a classification model.\"\"\"\n",
    "\n",
    "  bert_module = hub.Module(\n",
    "      BERT_MODEL_HUB,\n",
    "      trainable=True)\n",
    "  bert_inputs = dict(\n",
    "      input_ids=input_ids,\n",
    "      input_mask=input_mask,\n",
    "      segment_ids=segment_ids)\n",
    "  bert_outputs = bert_module(\n",
    "      inputs=bert_inputs,\n",
    "      signature=\"tokens\",\n",
    "      as_dict=True)\n",
    "\n",
    "  # Use \"pooled_output\" for classification tasks on an entire sentence.\n",
    "  # Use \"sequence_outputs\" for token-level output.\n",
    "  output_layer = bert_outputs[\"pooled_output\"]\n",
    "\n",
    "  hidden_size = output_layer.shape[-1].value\n",
    "\n",
    "  # Create our own layer to tune for politeness data.\n",
    "  output_weights = tf.get_variable(\n",
    "      \"output_weights\", [num_labels, hidden_size],\n",
    "      initializer=tf.truncated_normal_initializer(stddev=0.02))\n",
    "\n",
    "  output_bias = tf.get_variable(\n",
    "      \"output_bias\", [num_labels], initializer=tf.zeros_initializer())\n",
    "\n",
    "  with tf.variable_scope(\"loss\"):\n",
    "\n",
    "    # Dropout helps prevent overfitting\n",
    "    output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)\n",
    "\n",
    "    logits = tf.matmul(output_layer, output_weights, transpose_b=True)\n",
    "    logits = tf.nn.bias_add(logits, output_bias)\n",
    "    log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
    "\n",
    "    # Convert labels into one-hot encoding\n",
    "    one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)\n",
    "\n",
    "    predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))\n",
    "    # If we're predicting, we want predicted labels and the probabiltiies.\n",
    "    if is_predicting:\n",
    "      return (predicted_labels, log_probs)\n",
    "\n",
    "    # If we're train/eval, compute loss between predicted and actual label\n",
    "    per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)\n",
    "    loss = tf.reduce_mean(per_example_loss)\n",
    "    return (loss, predicted_labels, log_probs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qpE0ZIDOCQzE"
   },
   "source": [
    "Next we'll wrap our model function in a `model_fn_builder` function that adapts our model to work for training, evaluation, and prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FnH-AnOQ9KKW"
   },
   "outputs": [],
   "source": [
    "# model_fn_builder actually creates our model function\n",
    "# using the passed parameters for num_labels, learning_rate, etc.\n",
    "def model_fn_builder(num_labels, learning_rate, num_train_steps,\n",
    "                     num_warmup_steps):\n",
    "  \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
    "  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument\n",
    "    \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
    "\n",
    "    input_ids = features[\"input_ids\"]\n",
    "    input_mask = features[\"input_mask\"]\n",
    "    segment_ids = features[\"segment_ids\"]\n",
    "    label_ids = features[\"label_ids\"]\n",
    "\n",
    "    is_predicting = (mode == tf.estimator.ModeKeys.PREDICT)\n",
    "    \n",
    "    # TRAIN and EVAL\n",
    "    if not is_predicting:\n",
    "\n",
    "      (loss, predicted_labels, log_probs) = create_model(\n",
    "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
    "\n",
    "      train_op = bert.optimization.create_optimizer(\n",
    "          loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu=False)\n",
    "\n",
    "      # Calculate evaluation metrics. \n",
    "      def metric_fn(label_ids, predicted_labels):\n",
    "        accuracy = tf.metrics.accuracy(label_ids, predicted_labels)\n",
    "        f1_score = tf.contrib.metrics.f1_score(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        auc = tf.metrics.auc(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        recall = tf.metrics.recall(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        precision = tf.metrics.precision(\n",
    "            label_ids,\n",
    "            predicted_labels) \n",
    "        true_pos = tf.metrics.true_positives(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        true_neg = tf.metrics.true_negatives(\n",
    "            label_ids,\n",
    "            predicted_labels)   \n",
    "        false_pos = tf.metrics.false_positives(\n",
    "            label_ids,\n",
    "            predicted_labels)  \n",
    "        false_neg = tf.metrics.false_negatives(\n",
    "            label_ids,\n",
    "            predicted_labels)\n",
    "        return {\n",
    "            \"eval_accuracy\": accuracy,\n",
    "            \"f1_score\": f1_score,\n",
    "            \"auc\": auc,\n",
    "            \"precision\": precision,\n",
    "            \"recall\": recall,\n",
    "            \"true_positives\": true_pos,\n",
    "            \"true_negatives\": true_neg,\n",
    "            \"false_positives\": false_pos,\n",
    "            \"false_negatives\": false_neg\n",
    "        }\n",
    "\n",
    "      eval_metrics = metric_fn(label_ids, predicted_labels)\n",
    "\n",
    "      if mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        return tf.estimator.EstimatorSpec(mode=mode,\n",
    "          loss=loss,\n",
    "          train_op=train_op)\n",
    "      else:\n",
    "          return tf.estimator.EstimatorSpec(mode=mode,\n",
    "            loss=loss,\n",
    "            eval_metric_ops=eval_metrics)\n",
    "    else:\n",
    "      (predicted_labels, log_probs) = create_model(\n",
    "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
    "\n",
    "      predictions = {\n",
    "          'probabilities': log_probs,\n",
    "          'labels': predicted_labels\n",
    "      }\n",
    "      return tf.estimator.EstimatorSpec(mode, predictions=predictions)\n",
    "\n",
    "  # Return the actual model function in the closure\n",
    "  return model_fn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OjwJ4bTeWXD8"
   },
   "outputs": [],
   "source": [
    "# Compute train and warmup steps from batch size\n",
    "# These hyperparameters are copied from this colab notebook (https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\n",
    "BATCH_SIZE = 32\n",
    "LEARNING_RATE = 2e-5\n",
    "NUM_TRAIN_EPOCHS = 1.0\n",
    "# Warmup is a period of time where hte learning rate \n",
    "# is small and gradually increases--usually helps training.\n",
    "WARMUP_PROPORTION = 0.1\n",
    "# Model configs\n",
    "SAVE_CHECKPOINTS_STEPS = 1\n",
    "SAVE_SUMMARY_STEPS = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "emHf9GhfWBZ_"
   },
   "outputs": [],
   "source": [
    "# Compute # train and warmup steps from batch size\n",
    "num_train_steps = int(len(train_features) / BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
    "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oEJldMr3WYZa"
   },
   "outputs": [],
   "source": [
    "# Specify outpit directory and number of checkpoint steps to save\n",
    "run_config = tf.estimator.RunConfig(\n",
    "    model_dir=OUTPUT_DIR,\n",
    "    save_summary_steps=SAVE_SUMMARY_STEPS,\n",
    "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "colab_type": "code",
    "id": "q_WebpS1X97v",
    "outputId": "1648932a-7391-49d3-8af7-52d514e226e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './output', '_tf_random_seed': None, '_save_summary_steps': 1, '_save_checkpoints_steps': 1, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fef748916d8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using config: {'_model_dir': './output', '_tf_random_seed': None, '_save_summary_steps': 1, '_save_checkpoints_steps': 1, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
      "graph_options {\n",
      "  rewrite_options {\n",
      "    meta_optimizer_iterations: ONE\n",
      "  }\n",
      "}\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fef748916d8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
     ]
    }
   ],
   "source": [
    "model_fn = model_fn_builder(\n",
    "    num_labels=len(label_list),\n",
    "    learning_rate=LEARNING_RATE,\n",
    "    num_train_steps=num_train_steps,\n",
    "    num_warmup_steps=num_warmup_steps)\n",
    "\n",
    "estimator = tf.estimator.Estimator(\n",
    "    model_fn=model_fn,\n",
    "    config=run_config,\n",
    "    params={\"batch_size\": BATCH_SIZE})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NOO3RfG1DYLo"
   },
   "source": [
    "Next we create an input builder function that takes our training feature set (`train_features`) and produces a generator. This is a pretty standard design pattern for working with Tensorflow [Estimators](https://www.tensorflow.org/guide/estimators)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1Pv2bAlOX_-K"
   },
   "outputs": [],
   "source": [
    "# Create an input function for training. drop_remainder = True for using TPUs.\n",
    "train_input_fn = bert.run_classifier.input_fn_builder(\n",
    "    features=train_features,\n",
    "    seq_length=MAX_SEQ_LENGTH,\n",
    "    is_training=True,\n",
    "    drop_remainder=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t6Nukby2EB6-"
   },
   "source": [
    "Now we train our model! For me, using a Colab notebook running on Google's GPUs, my training time was about 14 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "nucD4gluYJmK",
    "outputId": "5d728e72-4631-42bf-c48d-3f51d4b968ce"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beginning Training!\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'datetime' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-6187edf75f66>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf'Beginning Training!'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcurrent_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mestimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_fn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtrain_input_fn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mnum_train_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Training took time \"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdatetime\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mcurrent_time\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'datetime' is not defined"
     ]
    }
   ],
   "source": [
    "print(f'Beginning Training!')\n",
    "current_time = datetime.now()\n",
    "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
    "print(\"Training took time \", datetime.now() - current_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CmbLTVniARy3"
   },
   "source": [
    "Now let's use our test data to see how well our model did:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JIhejfpyJ8Bx"
   },
   "outputs": [],
   "source": [
    "test_input_fn = run_classifier.input_fn_builder(\n",
    "    features=test_features,\n",
    "    seq_length=MAX_SEQ_LENGTH,\n",
    "    is_training=False,\n",
    "    drop_remainder=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 445
    },
    "colab_type": "code",
    "id": "PPVEXhNjYXC-",
    "outputId": "dd5482cd-c558-465f-c854-ec11a0175316"
   },
   "outputs": [],
   "source": [
    "estimator.evaluate(input_fn=test_input_fn, steps=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ueKsULteiz1B"
   },
   "source": [
    "Now let's write code to make predictions on new sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OsrbTD2EJTVl"
   },
   "outputs": [],
   "source": [
    "def getPrediction(in_sentences):\n",
    "    labels = [\"Negative\", \"Positive\"]\n",
    "    input_examples = [run_classifier.InputExample(guid=\"\", text_a = x, text_b = None, label = 0) for x in in_sentences] # here, \"\" is just a dummy label\n",
    "    input_features = run_classifier.convert_examples_to_features(input_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
    "    predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=False)\n",
    "    predictions = estimator.predict(predict_input_fn)\n",
    "    return [(sentence, prediction['probabilities'], labels[prediction['labels']]) for sentence, prediction in zip(in_sentences, predictions)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-thbodgih_VJ"
   },
   "outputs": [],
   "source": [
    "pred_sentences = [\n",
    "  \"That movie was absolutely awful\",\n",
    "  \"The acting was a bit lacking\",\n",
    "  \"The film was creative and surprising\",\n",
    "  \"Absolutely fantastic!\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 649
    },
    "colab_type": "code",
    "id": "QrZmvZySKQTm",
    "outputId": "3891fafb-a460-4eb8-fa6c-335a5bbc10e5"
   },
   "outputs": [],
   "source": [
    "predictions = getPrediction(pred_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MXkRiEBUqN3n"
   },
   "source": [
    "Voila! We have a sentiment classifier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "colab_type": "code",
    "id": "ERkTE8-7oQLZ",
    "outputId": "26c33224-dc2c-4b3d-f7b4-ac3ef0a58b27"
   },
   "outputs": [],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Predicting Movie Reviews with BERT on TF Hub.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "conda_tensorflow_p36",
   "language": "python",
   "name": "conda_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
